System and method for continuous media segment identification

ABSTRACT

This invention provides a means to identify unknown media programming using the audio component of said programming. The invention extracts audio information from the media received by consumer electronic devices such as smart TVs and TV set-top boxes then conveys said information to a remote server means which will in turn identify said audio information of unknown identity by way of testing against a database of known audio segment information. The system identifies unknown media programming in real-time such that time-sensitive services may be offered such as interactive television applications providing contextually related information or television advertisement substitution. Other uses include tracking media consumption among many other services.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional PatentApplication No. 62/086,113, entitled “AUDIO MATCHING USING PATHPURSUIT,” filed Dec. 1, 2014, naming W. Leo Hoarty as the inventor. Theforegoing application(s) are either currently co-pending or areapplications of which the instant or a currently co-pending applicationis entitled to the benefit of the filing date.

FIELD OF THE INVENTION

The present invention relates generally to a media identification clientserver system with significant improvements in efficiently representingand identifying multimedia information. More particularly, the presentinvention addresses a computationally efficient and accurate mediaidentification system requiring only minimal processing of media at theclient device process prior to communicating to server means forcontinuous identification.

BACKGROUND

Applications for automated content recognition are experiencingconsiderable growth and are expected to continue to grow fueled bydemand from many new commercial opportunities including: interactivetelevision applications providing contextually related content; targetadvertising; and, tracking media consumption. To address this growth,there is a need for a comprehensive solution related to the problem ofcreating a media database and identifying, within said database, aparticular media segment that is tolerant of media content alterationssuch as locally-generated graphics within the client device altering theoriginally transmitted picture or a user watching a standard definitionbroadcast while using the zoom or stretch mode of their HDTV. Thesealterations can occur due to user actions such as engaging an electronicprogram guide (EPG, requesting additional program information that thenappears in a set-top-generated pop-up window or selecting a non-standardvideo mode on a remote.

Automated content recognition systems typically ingest considerablequantities of data and often operate on continuous round-the-clockschedules. The amount of data consumed and managed by said systemsqualifies them to be classified by the currently popular idiom ofbig-data systems. It is therefore imperative that said systems operateas efficiently as possible in regards to both data processing andstorage resources as well as with data communications requirements. Afundamental means to increase operational efficiency while stillachieving requisite accuracy is to utilize a method of generating acompressed representation of the data to be identified. Said compressedrepresentations are often called fingerprints which are generallyassociated with identifying data from the audio or video content.Although a diverse range of algorithms of varying complexity are used,most rely on a common set basic principles which have several importantproperties such as: the fingerprint should be much smaller than theoriginal data; a group of fingerprints representing a media sequence ormedia segment should be unique such that said group can be identified ina large database of fingerprints; the original media content should notbe able to be reconstructed even in a degraded form from a group offingerprints; and, the system should be able to identify copies oforiginal media even when said copies are diminished or distortedintentionally or by any means of copying or otherwise reproducing saidmedia. Examples of common media distortions include: scaling or croppingimage data such as changing from a high-definition video format to astandard definition format or vice-versa, re-encoding the image or audiodata to a lower quality level or changing a frame rate of video. Otherexamples might include decoding digital media to an analog form thendigitally re-encoding said media.

A useful example of a typical media fingerprint process can beillustrated by examining the popular mobile phone application (app)called ‘Shazam.’ The Shazam app and many similar apps are typically usedto identify a song unknown to the user particularly when heard in apublic place such as a bar or restaurant. These apps sample audio fromthe microphone of a mobile device such as a smartphone or tablet andthen generate what is known as a ‘fingerprint’ of the unknown audio tobe identified. Said ‘fingerprint’ is generally constructed by detectingfrequency events such as the center frequency of a particular soundevent above the average of surrounding sounds. This type of acousticevent is called a ‘landmark’ in the Shazam patent U.S. Pat. No.6,990,453. The system then proceeds to analyze the audio for anothersuch event. When found the first ‘landmark’ and the second ‘landmark’along with the time interval separating them are sent as a data unitcalled a ‘fingerprint’ to a remote processing means to be accumulatedwith additional ‘fingerprints’ for a period of time, usually twenty tothirty seconds. The series of ‘fingerprints’ are then used to search areference database of known musical works where said database wasconstructed by said fingerprinting means. The match result is then sentback to the mobile device and, when the match result is positive,identifies the unknown music playing at the location of the user.

Another service, called Viggle identifies TV audio by means of asoftware app downloaded to the user's mobile device which relays samplesof audio from the user's listening location to a central server meansfor the purpose of identifying said audio by means of an audio matchingsystem. is The service provides means for users of the service toaccumulate loyalty points upon identification of TV programs while saidusers watch said programs. The service user can later redeem saidloyalty points for merchandise or services similar to other consumerloyalty programs.

The identification of unknown television segments generally requiresvery different processes between the identification of video and theidentification of audio. This is due to the fact that video is presentedin discreet frames and audio is played as a continuous signal. However,in spite of differences in presentation format, said video systemscompress video segments to representative fingerprints and then search adatabase of known video fingerprints in order to identify said unknownsegment similar to the identification process of audio. Said videofingerprints can be generated by many means but generally the primaryfunction of fingerprint generation requires the identification ofvarious video attributes such as finding image boundaries such as lightto dark edges in a video frame or other patterns in the video that canbe isolated and tagged then grouped with similar events in adjacentvideo frames to form the video fingerprint.

In principle, systems that identify video segments should be built usingthe same processes to enroll known video segments into a referencedatabase as used to process unknown video from a client means of a mediamatching service. However, using the example of a smart TV as saidclient means, several problems arise with sampling the video arriving atthe television using the processing means of the smart TV. One suchproblem arises from the fact that the majority of television devices areconnected to some form of set-top device. In the United States, 62% ofhouseholds subscribe to cable television service, 27% subscribe tosatellite TV and a growing number of TV are fed from Internet connectedset-tops. Less than 10% of television receivers in the U.S. receivetelevision signal from off-air sources. In the case of set-topsproviding television signals to the television set, as opposed toviewing television from off-air transmissions via an antenna, theset-top will often overlay the received video picture with a locallygenerated graphic display such as program information when a userpresses an ‘Info’ button on the remote control. Similarly, when the userrequests a program guide, the TV picture will be typically shrunk to aquarter-size or less and positioned in a corner of the displaysurrounded by the program guide grid. Likewise, alerts and othermessages generated by a set-top can appear in windows overlaying thevideo program. Other forms of disruptive video distortion can occur whenthe user chooses a video zoom mode which magnifies the picture or astretch mode when the user is viewing a standard definition broadcastbut wishes the 4:3 aspect ratio picture to fill a high-definitiontelevision 16:9 screen. In each of these cases, the video identificationprocess will fail in matching the unknown video sampled from saidset-top configurations.

Hence, existing automated content recognition systems that rely on onlyvideo identification will be interrupted when a number of commonscenarios arise, as outlined above, that alter the video programinformation by an attached set-top device. Yet further problems arisewith identifying video even when video is not altered by a set-topdevice. For example, when a video picture fades to black or even whenthe video image is portraying a very dark scene, the prior art of videoidentification systems can lose the ability to identify the unknownvideo segment.

Interestingly, the audio signal of a television program is almost neveraltered but conveyed to the television system as received by a set-topdevice attached to said TV. In all of the above examples of graphicsoverlays, of fades to black or dark video scenes, the program audio willcontinue to play usually unaltered and hence be available for reliableprogram segment identification by means of a suitable automated contentrecognition system for audio signals. Hence, there is a clear need foran automated content recognition system that utilizes audioidentification either alone or in addition to identifying video for thepurposes of identifying unknown television program segments. However,the technology employed by the above mentioned music identificationsystems, such as Shazam, are not generally suited for identification ofcontinuous content such as a television program. These mobile phonemusic identification apps are typically designed to process audio from amicrophone exposed to open air which also imports significant room noiseinterference such as found in a noisy restaurant or bar. Also, the modeof operation of these above-mentioned audio identification applicationsis typically based on presumptive ad hoc usage and not designed forcontinuous automated content recognition. Hence, because of the manytechnical challenges of identifying audio from high interferencesources, the technical architecture of ad hoc music ID programs is notsuitable for continuous identification of audio. Said systems wouldsuffer further from operating not only continuously but with very largenumbers of simultaneous devices, such as a national or even regionalpopulation of television set-tops or smart TVs.

Many uses exist for identifying television programming as it isdisplayed on a television receiver. Examples include interactivetelevision applications where a viewer is supplied supplementalinformation to the currently displaying TV program often in the form ofa pop-up window on the same TV display from which media is identified oron a secondary display of a device such as a smartphone or tablet. Suchcontextually related information usually requires synchronization withthe primary programming currently being viewed. Another application ofdetecting television programming is advertisement substitution alsoknown as targeted advertising. Yet another use exists for media censussuch as audience measurement of one or more television programs. All ofthese uses and others not mentioned benefit from timely detection ofunknown program segments. Hence, continuous audio identification aloneor in concert with video identification can provide or enhance thereliability and consistency of an automated content recognition system.

SUMMARY OF THE INVENTION

The invention is used to identify video and/or audio segments for thepurposes of enabling interactive TV applications to provide variousinteractive television services in a client set-top box or smart TV. Inaddition, the invention provides a reliable means to identify programviewing statistics for audience measurement purposes.

The invention provides audio and video segment identification meanswhere upon enrollment, as illustrated in FIG. 1, frames of video as wellas seconds of audio are transformed into a common format of continuouscoefficient streams 101 that can be tagged and stored in a referencedatabase 102 for the purpose of providing candidate data for theidentification of unknown audio or video segments when presented to thesystem of the invention from a client device enabled by the invention.The invention can operate in multiple modes such as with only video orwith only audio or a combination of both video and audio and the systemwill provide accurate results within three to ten seconds. Audio andvideo segment information is prepared for the identification process ina manner 103 that is identical to the enrollment process 101 for theprocess of identification 104 of FIG. 1. The result of a successfulmatch is either a unique identification code or the metadata of theaudio/video segment 110.

In one embodiment of the invention, video segments may be utilized asthe primary means of identifying unknown media segments. If a consumerdevice such as a set-top box displays locally generated graphics thatoverlay the primary video picture, video identification by the inventionmight be interrupted. If said interrupting occurs, the system of theinvention can seamlessly switch to the audio segment information tocontinue identifying the unknown media content sent to the centralmatching server means from said consumer device.

The ability to dynamically switch between audio and video segmentidentification is further enhanced by an embodiment of the inventionwhere audio segment information is transformed by a Linear PredictiveCoding (LPC) means of the invention from a stream of digital audiosamples to a stream of coefficients or symbols with characteristicssimilar to the video segment transformation process. Saidcharacteristics include a broad set of symbols, called coefficients,that exhibit wide variability without a direct correlation to frequency,unlike other time-to-frequency transforms such as the well-known andpopular Fourier series. Furthermore, the said coefficients process willreliably repeat in values for the same or largely similar segments ofaudio, hence, exhibiting the very desirable characteristics of apparenthigh entropy while retaining repeatability. Another important feature ofthe LPC process of the invention is said coefficients values remainessentially stationary for time intervals of a minimum of 20milliseconds (ms) to as much as 100 ms. Said stationary time framesallow the coefficients to be treated with processing means similar tothe video pixel sampling process of Neumeier U.S. Pat. No. 8,595,781,incorporated herein in its entirety by reference, which provides thefurther advantage of allowing the use of continuous data matchingschemes employing high-dimensional algebraic suspect selection inconjunction with time-discounted scoring means such as Path Pursuit astaught by Neumeier. This is in sharp contrast to prior art where featurevectors and other means are used to find landmarks and landmarks arecombined to form fingerprints as exemplified by the popular Shazam musicidentification service and many other audio identification systems.

Audio data is considerably different from video data in most respectsyet the audio signal is transformed by the invention into sets or framesof coefficients, also known to the art as ‘cues’, in such a way as toresemble sampled pixel values of video information. This aspect of datasimilarity between video and audio cues allows the advantageous centralmatching means of the invention to be used interchangeably for eithermatching unknown audio against reference audio or unknown video againstreference video data or to process both simultaneously, if anapplication should require this.

The invention provides a means to continuously identify mediainformation from a plurality of client devices such as smart TVs, cableor satellite set-top boxes or Internet media terminals. The inventionprovides a means for samples of media received by said devices to betransformed into continuous frames of compressed media information foridentification by a central server means. Said central server means willidentify unknown media segments within three to ten seconds and providethe identity of the previous unknown segment back to the respectiveclient device that provided said segment for use in interactivetelevision applications such as the display of contextually relatedcontent in overlay windows, for instance, or for the purposes ofadvertisement substitution. Additionally, the identification of mediasegments can be supplied to other processes of the server, or externalsystems via a network, for media census such as audience measurementapplications.

The invention is based on the transforming of audio into time-frozenframes of coefficients in a continuous process that is similar to thecontinuous video frame processes of the prior art (Neumeier patent) andis accomplished by understanding that, in Neumeier, the videoinformation is processed by finding average pixel values from aplurality of video frame locations within a video frame. Said videoframe information is enrolled in the matching systems continuously,generally at a rate of at least multiple frames per second but notnecessarily the full video frame rate of ordinary television signals.Likewise, the identification phase of the Neumeier patent allows saidvideo frame information to be collected and transferred to the centralmatching means of the invention at video frame rates less than the fullframe rate of the unknown video segment as long as the frame rate is notgreater than the enrollment frame rate. The audio information isprocessed as overlapping frames of typically short duration audiosegments of typically 20 to 100 milliseconds. It is known that certainaudio channel characteristics such as the power spectral density of asignal is effectively stationary over short intervals of between 20 to100 milliseconds and can be converted to coefficients that do not changeappreciably within said frame time. Hence a means is available totransform continuous audio data into essentially time-frozen frames ofcoefficients that provide an efficient means to store known audioinformation in a database then later search by algorithmic means toidentify an unknown audio segment.

In addition, it has been determined in the process of development of theinvention that said coefficients have entropic characteristics similarto said video coefficients (cues) of patent U.S. Pat. No. 8,595,781providing the ability to store said coefficients by means of a localitysensitive hash indexing means to form a searchable reference database.As with video, during the identification phase, the database can besearched by linear algebraic (matrix mathematical) means to findcandidates in multidimensional space. Said candidates, also calledsuspects, can be represented by a token placed in a bin withcharacteristics resembling a leaky bucket providing an effective scoringmeans known in the art as time-discount binning to find a match resultfrom the harvested suspects. Yet another effective means to scorecandidate matches is by means of correlation of said unknown cue to oneor more candidate (known) cues. Said means of correlation, not to beconfused with auto-correlation as used herein, is well known to theskilled person for finding the closest match of a reference data item toone data item of a set of test data items. Hence, said scoring means bythe process of mathematical correlation produces a best match by theidentification system in place of time discount binning.

It should be understood that the coefficient frame generation rateduring the identification process can be less than the coefficient framegeneration rate used during the enrollment process as still providesufficient information for the matching system to accurately determinethe identity of an unknown audio segment in a three to ten second timeinterval. For example, the invention allows the enrollment rate tooperate at, say, 20 millisecond intervals (with 50% overlap, forexample) equaling 100 frame times per second. A client device couldtransmit frames to the matching server means for identification atperhaps 50, 25 or 10 frames per second or any reasonable multiple of 100in order for effective matching to occur by the identification mechanismof the invention.

Once audio is transformed from a time-based to a frequency-basedrepresentation, additional transformations may be applied in order togenerate certain further refinements to coefficient frame (cue) sets. Inthis step, one finds a diversity of applicable algorithms. The objectiveis to reduce the data dimensionality and, at the same time, to increasethe invariance to enrollment versus identification sample alignment.Hence, a multiplicity of coefficient generation capabilities existswhere any one of said coefficients can be chosen for use in dataenrollment and identification assuming only one specific choice is inapplied at any given time for both enrollment and for identification.

This invention provides a means to identify audio or video informationfrom any source of media such as cable, satellite or Internet deliveredprogramming. Once identified, the invention can send a signal from thecentralized identification means to a client application of theinvention by means of a data network causing said application to displaycontextually targeted or other content on a television displayassociated with the client device providing the unknown mediainformation. Likewise, said contextually coordinated content can besupplied by said identification means to a second screen device such asa smartphone or tablet. Similarly, upon identification of an unknownmedia segment, the invention can maintain a viewing census for audiencemeasurement of specific television programming for use by third-partiessuch as television advertisement agencies or television networks.

In one or more various aspects, related systems include but are notlimited to circuitry and/or programming for effecting theherein-referenced method aspects; the circuitry and/or programming canbe virtually any combination of hardware, software, and/or firmwareconfigured to effect the herein-referenced method aspects depending uponthe design choices of the system designer.

In addition to the foregoing, various other methods, systems and/orprogram product embodiments are set forth and described in the teachingssuch as the text (e.g., claims, drawings and/or the detaileddescription) and/or drawings of the present disclosure.

The foregoing is a summary and thus contains, by necessity,simplifications, generalizations and omissions of detail; consequently,those skilled in the art will appreciate that the summary isillustrative only and is NOT intended to be in any way limiting. Otheraspects, embodiments, features and advantages of the device and/orprocesses and/or other subject matter described herein will becomeapparent in the teachings set forth herein.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a top-level block diagram of the basic functions of anautomated content recognition system. Known audio/video information 101consisting of audio and/or video segments 102 and metadata (programinformation) 103 is processed and transformed into coefficient frames104 which is stored in a reference database 105. Unknown audio and/orvideo information 106 is processed into coefficient frames 107 by meansof a similar process as 104 and supplied to an Automated ContentRecognition (ACR) system 108 which compares the data against saidreference database 105. When said unknown audio/video segment isidentified, audio and/or video metadata (program information or segmentID) is output 109.

FIG. 2 is a block diagram of the server 202 and client 203 means of theinvention. One or more content sources 201 a are supplied to a MediaIngest means 201 which produces Audio and/or Video Cue data 201 c aswell as providing associated metadata in the form of ProgramIdentification and Timecode 201 b information for each of the mediasegments. Said media information is entered into a Reference MatchDatabase 204 which is queried by Automated Content Recognition (ACR)Processor 205 to process and identify unknown audio 203 b and/or video203 a segments as supplied by one or more client devices 203. Saidclient device consists of an ACR Client 208 which converts the contentsof a television frame buffer 209 and/or television audio buffer 211 intorespective cue sets which are sent to server 202. Upon successfullymatching an audio or video segment, ACR Processor 205 sends a message toMatch Processing means 207 which thereby examines an InteractiveTelevision (ITV) Content Database for the presence of instructions andpossibly data to be transmitted by network to client device applications210 for local processing by client device 203. Said processing mayinclude the display of supplemental information in a window on atelevision display with information associated with the program segmentdetected by the process of the invention. Additionally, Match Processing207 may provide results to a measurement database such as an audiencemeasurement system 207 b.

FIG. 3 is a block diagram of an advantageous system that illustrates ameans for the invention to receive media information such as radio ortelevision programming broadcast from a Content Delivery Network 302via, for example, optical transmission means 303 such that the MatchingServer System 306 will receive said programming in advance of the clientdevices, such as a smart TV, such that the content can be processed andstored in a Reference Media Database 307 with sufficient time such thatthe system is ready ahead of the arrival of unknown media from ClientDevices 309 to 312. The network distribution of radio or televisionprogramming is often provided to service providers such as satellite andcable TV providers by means of fiber optic networks which typicallyexhibits network latencies of fractions of one second whereas the clientdevices may receive content via satellite or by said content passingthrough additional processing in the headend of a cable system such thata delay is incurred of about two to five seconds or possibly longer.This difference in distribution time between the backbone and the homedelivery is sufficient to allow the server means of the invention toprovide real-time processing of unknown audio or video segments as theknown data from the same sources as received by said client devices willhave already been processed and stored for use by said Matching Servermeans well in advance of any queries of its matching service. Hence,interactive TV services such as contextually-related information displayor advertisement substitution can be carried out very near the start ofthe playout of the identified segment.

FIG. 4 is a flow diagram of the processing of raw audio input 401 from areceiver showing the steps of preprocessing 402; pre-emphasis (ifapplied) 403; framing, shaping & overlapping of audio segments 404;autocorrelation 405 to prepare the signal for the process of LinearPredictive Coding 406; then LPC coefficient transformation into eitherLine Spectral Pairs or Immittance Spectral Frequencies 407; thenpost-processing of coefficients by means of normalization & quantization408; and formation of quantized coefficients into ‘cues’ sets 409 to betransmitted to an Audio Matching System 410 which provides AudioMetadata (Identification) 411 when an audio segment is successfullyidentified by said matching system.

FIG. 5 is a graph of the frequency response of an audio pre-emphasisfilter to enhance the information content of high-frequency audiocomponents;

FIG. 6 is a plot (a) of typical television audio spectrum before thepre-emphasis filter of FIG. 5 is applied to said signal. A measurementof the difference in amplitude of the audio signal from low-frequencyaverage peak (around 500 Hz) to high-frequency average-peak 601 shows arange of approximately 45 dB. Plot (b) shows the increased signalstrength of high frequency audio components after plot (a) is processedthrough filter of FIG. 5 with the high-frequency information increasedto a beneficially narrower range of 30 dB 602 between said frequencies.

FIG. 7 illustrates an audio segment overlap 701 to 704 as employed bythe invention. In one embodiment, the invention uses 20 millisecondaudio segments with a 10 millisecond overlap. In certain embodiments,segment lengths can beneficially utilize segment lengths up to 100milliseconds and overlaps can be beneficially realized from 10% to 90%of the segment length.

FIG. 8 is a plot of signal framing showing the spectral effects ofvarious shaping functions as applied to an audio frame. The graph 801shows a simple Rectangular Frame with an abrupt start and stop resultingin Fourier transform 802 showing significant sideband noise added to thesignal of interest as a result of the sudden discontinuities. Plot 803illustrates a Hamming Window widely used in voice communicationssystems. The resulting Fourier transform 804 shows an optimized signalwith harmonic information suppressed by >50 dB. Plot 805 shows arelatively simple Triangular Window Function which has a Fourier plot806 close in quality to the Hamming window plot 804 but requires farless computation to apply to the audio frame and, hence, is mostadvantageous for application with consumer electronics devices such assmart TVs or set-top boxes with limited computational means.

FIG. 9 is a plot of the coefficient output of an autocorrelationfunction as employed by the invention and applied to typical televisionaudio.

FIG. 10 is a plot of a Linear Predictive (LP) spectrum 1002 illustratedwith a plot of a weighting filter 1001 appropriate to normalize thecoefficients for optimal quantization.

FIG. 11 is a plot of the coefficient output of an LPC process of theautocorrelation output of FIG. 10 showing typical values of a 20 msecaudio sample of a speech signal.

FIG. 12 Result of LPC coefficient output of FIG. 11 transformed toImmittance Spectral Frequencies (ISF) coefficients. It is well known tothe art that a suitable alternative exists employing the Line SpectralPairs (LSP) transform which produces similar coefficients where both ISFand LSP coefficients can be more suitable for quantization that theunprocessed coefficients of the LPC process.

FIG. 13 is a polar plot of ISF Coefficient mapping of the coefficientoutput of the ISF process to the complex-plane (Z plane) unit circle.The ISF coefficients exist in symmetrical conjugate pairs and only thefirst half of the unit circle contribute to the output values. The poles(x's) of the LPC that formed the input to the ISF process are shownwithin the circle.

FIG. 14 is a chart of 15 of ISF coefficients graphed over time showingrelative sensitivity of unmodified transformed outputs relative toposition 1203 on the unit circle of a Z-axis plot.

FIG. 15 Ingest process of audio source 1501 decoded intoreceiver/decoder audio buffer 1502, then segmented into audio frames ofa fixed length 1503. Audio frames are transformed 1504 by means of, inthis embodiment, autocorrelation then further processed by LinearPredictive Coding 1505 into coefficients 1505 and yet further processed1506 into coefficients, in this embodiment, using the ISF transform.Program information metadata 1509 is added to program time-code 1508 tothe processed coefficients 1507 to form an audio data cue record 1510.

FIG. 16 Diagram of reference audio cue 1601 as hashed by Audio HashFunction 1602 and stored in reference database 1604 indexed by parsingthe output of said Hash Function 1602 with most significant bitsaddressing a storage sector and the remaining bits addressing a “bucket”(location) 1606 within said storage sector.

FIG. 17 Diagram of audio cue formation 1706 from an unknown audio sourceas received by Television Monitor 1701 and decoded in said TV AudioBuffer 1703 then processed by client software of the invention to formAudio Frames of predetermined length 1702 and transformed tocoefficients 1705. Said client side cue formation included the additionof the current time of processing 1707 known to the art as “wall time”.

FIG. 18 Diagram of Unknown Audio Cue 1801 generating a hash index bymeans of Hash Function 1804 then used to address reference databaseBucket 1805. Candidate Audio Cues 1802 are retrieved from said databaseand supplied to Matching Process 1803 which output result 1807 upon asuccessful matching of unknown media segment to known segment fromreference database 1806.

FIG. 19 is a representative diagram of the Time Discount Binning process1901 that supplies tokens to buckets 1902 until a bucket containssufficient tokens to cross threshold 1904 indicating a high-probabilityof, in the invention, a media segment matching result. Said buckets are“leaky” and will drain tokens over time such that consistent results ofmatches are required within a predetermined time domain to cause tokensto fill respective buckets faster than the rate of leakage in order fortokens in said bucket to successfully cross said threshold.

FIG. 20 is a matrix diagram of possible combinations of thetransformation from audio input to coefficient or hash string output. Inall paths through said matrix, with the exception of output 2013, thecoefficients are quantized by either a linear process 2014 or by meansof vector quantization 2015 then output from the system at 2016. In allof these processes, audio is transformed into high-entropy coefficientsets representing frames of audio with near stationary power spectrumfor the duration of the audio frame hence generating coefficient thatcan be appropriately hash indexed and applied to a search and scoringmeans of Path Pursuit for the continuous identification of audiosegments.

FIG. 21 This flow chart includes steps in which content audio matchingmay be performed.

FIG. 22 This flow chart defines the steps of matching a series ofcoefficient frames representing an unknown audio segment. The candidateharvesting (determination) and the time-discount binning is the same astaught by Neumeier patent.

FIG. 22a This flow chart defines the steps of matching a series ofcoefficient frames representing an unknown audio segment. The candidateharvesting (determination) is supplied to a process of correlation ofthe unknown cue set to one or more suspect (candidate) cues. The closestmatch is further evaluated and if above a threshold is then output asthe result.

FIG. 23 illustrates an operational flow representing example operationsrelated to continuous audio matching.

FIGS. 24 to 28 illustrate alternative embodiments of the operationalflow of FIG. 23.

DETAILED DESCRIPTION OF THE INVENTION

In one embodiment, as illustrated in FIG. 2, the system identifies audio203 b and video 203 a information from television programming by meansof a client application 203 of the invention operating within theprocessor means of a cable TV, satellite or Internet-connected set-topbox or within the processor means of a smart TV. In an exampleembodiment, said client application process typically operates on theaudio 211 and/or video 209 information just prior to said informationplaying to the speakers and/or display of said television device. Saidaudio and/or video information is processed by the invention to producea highly-compressed, continuous stream of frame representations of therespective audio and/or video signal by means of ACR Client 208. Saidframe representations are transmitted 203 a and/or 203 b via a network,typically the Internet, to a server means 202 of the invention foridentification. Said frame representations are of the form of selectaveraged pixel values for video frames and transformed power spectralcoefficients for audio information.

In order to identify unknown media segments of audio and/or videoinformation, said information must first be enrolled by theidentification server means of the invention 104 and 105 of FIG. 1. Saidenrollment process is typically the same or similar to the processrendered by a client device 107 to send said coefficient representationto said server 108. Said enrollment data is received by the server 102,processed and then stored by the server at 105 for later utilization bythe identification process 108.

Referring again to FIG. 2, upon successfully identifying the unknownmedia segment at ACR Processor 205, the system of the invention cansearch a process of the server by means of Match Processing 207 to finda client service in ITV Content Database 206 that may be notified ortriggered by the presence of the media segment. Said client event mayinclude transmitting a trigger signal 202 a to a client application 210of the invention that displays contextually related information such asinformation about the program plot or an actor in the program or any ofa variety of interactive television services available from the smart TVor set-top box. Likewise, said trigger could cause a currentlydisplaying television advertisement to be substituted for a differentadvertisement that is more relevant to the viewer. Said ad substitutionprocess is also known to the skilled person as targeted advertising. Yetanother use of said trigger is to update a viewership database via 207 bto maintain a viewing census for audience measurement purpose. Saidcensus is typically less time sensitive that the other interactive TVuses described above.

Audio and video match data streams are created by separate and distinctprocesses however each process results in data structures of similarcharacteristics which may then be applied to separate databases yetserviced by equivalent server means of the invention for both enrollmentof the data into a reference database as well as for use by the mediamatching means of the invention for identification of unknown mediasegments from client devices. Video and audio coefficients, thoughsomewhat similar in characteristics of dimensionality and entropy, aremaintained in separate databases and it should be obvious to the skilledperson that audio data cannot be used to search a video database andvice versa. However, the processing means and database structures aresimilar and are largely the same for both types of media hence affordingan advantageous economy of scale for systems employing both video andaudio matching.

Video coefficients are generated from video information as taught by theinvention of patent U.S. Pat. No. 8,595,781. The searchable audiorepresentations of the invention must be formed from a very differenttype of media than video information. However, the end result of theprocess is a continuous stream of coefficients frames that have thesimilar characteristics to the video frame information as created bysaid referenced patent.

For the creation of searchable frames of audio coefficients from audioinformation, it is a fundamental aspect of the invention that the powerspectral density of a typical audio signal such as television audioremains essentially stationary for a period of 20 to as much as 100milliseconds (msec) which is in range of a single television frame ofapproximately 33 milliseconds for U.S.-based standards and 40milliseconds for the non-U.S.-based television. Hence, an audio signalcan be segmented into frames and then converted to a power spectralrepresentation and stored in a searchable multi-dimensional referencedatabase with a process similar to video frames, as taught by Neumeier,from which a subset of pixels is sampled and stored in a match database.One embodiment of this invention that provides the necessary audio datatransformation employs the use of Linear Predictive Coding (LPC) as theprimary step to convert an audio signal into said audio coefficientrepresentations to then be transmitted to the server of the invention.The use of LPC or an equivalent transform allows for flexible andefficient transformation of the audio signal into a highly compressedform that can be further manipulated to enhance the search and selectionefficiency of the overall system of automated content recognition.

In contrast, the prior art for audio matching may convert, for example,an audio signal from a time to frequency representation using, forexample, a Modified Discreet Cosine Transform (MDCT), a Med FrequencyCepstral Coefficient (MFCC) process or a Discreet Fourier Transform,etc. Once the signal is converted, the prior art may find frequencyevents above a particular magnitude, sometimes called landmarks, andthen measures the time interval between events or landmarks to form socalled fingerprints for storing reference media segments. The sameprocess is then used by a client device for producing fingerprints to besubmitted to identify unknown media segments.

For the purposes of matching audio information, the invention does notuse the fingerprint means of the prior art but rather creates continuousstreams of coefficient from fixed frames of audio for building areference database and then, for matching unknown media segments, asimilar process is applied by a client device to an unknown audiosegment and said coefficients are supplied to a matching server meansutilizing said reference database. It should be understood that thecoefficient process of the invention can be realized by a variety ofdifferent but related mathematical transforms as charted in FIG. 20which are somewhat similar to those used by prior art. However, the manyadditional steps by the prior art in the formation of fingerprintsconstructed from identifying landmarks or other unique constructs is notin any way utilized by the invention. Hence, the invention is able tooperate on continuous streams of media where the prior art cannot.Additionally, the invention is massively scalable to supporting millionsof client devices with high accuracy and the further advantage of lowprocessing overhead in the client device.

Returning to FIG. 2 of the invention showing the client to server basicfunctions and communications paths, a client device 203 contains aprocessor means capable of executing computer programs and client deviceprovides access to said processor means to the video 209 and audio 211buffers of said client. An ACR Client 208 application periodicallysamples data from said video and audio buffers and processes video 203 aand audio 203 b cues where a cue is composed of the elements of FIG. 171706. In this embodiment, the elements of a cue consist of 16coefficients and time-code consisting of the local time (also known aswall time). Said cues are transmitted via a network to the server meansof the invention 202. An automated content recognition (ACR) processor205 receives said cues and performs a matching process where receivedcues are identified by means of searching reference media match database204. Said processor 205 can provide useful match results by a variety ofmeans, for example, by the use of Path Pursuit of Neumeier or by meansof the correlation of an unknown cue set to a set of suspect cues. Thecorrelation process is disgrammed in FIG. 22a . Positive identificationfrom 205 is conveyed to a match processing means 207 which can execute avariety of functions such as providing contextually related content tothe client device as taught by patent U.S. Pat. No. 8,769,584 B2 of ZeevNeumeier, incorporated herein in its entirety by reference. The matchingprocessing 207 can also provide statistical information to match resultsservice 207 b for audience measurement purposes or other audiencemeasurement services.

FIG. 3 shows how the invention has the ability to provide continuousidentification of, for example, television programming. Many interactivetelevision applications are made possible by a system that has timelyknowledge of the current program displayed on a television receiver.Such applications include targeted advertising as well as contextuallytrigger information displays. Though not necessarily time sensitive,accurate audience measurement is also enabled by the system of theinvention. FIG. 1 shows media information processed by the enrollmentsystem in order to populate a reference database against which unknownmedia information is tested for identification. The obvious problem ishow to get data, such as television programming, into a central databasequickly enough that the same television programming entering the systemfrom the client device can be matched without delay. The answer lies inthe fact that the central enrollment system received media content fromthe television distribution backbone which arrives at the central meansof the invention usually four to ten seconds ahead of the sameprogramming arriving at the television receiver of the client device.Hence the system has sufficient time to process incoming reference mediaahead of any queries requiring said data.

In a preferred embodiment of the invention, FIG. 4 depicts the steps ofconverting a client television receiver audio 401 into data suitable fortransmission to an audio matching system 410. The process of saidtransformation begins with the audio pre-processing function 402 wheredigital audio received from the audio buffer of a television receivingdevice is converted from stereo to monaural by means of summing saidstereo information and may be further processed by a down-sampling stepwhere, in one embodiment, said digital audio may be provided at a highersample rate, for example 48 kHz but is to be processed by the inventionat, for example, 16 kHz. Other preprocessing steps may include volumenormalization and band filtering. Process 403 applies a process ofpre-emphasis where the audio signal is passed through a high pass filterwith the filter characteristics shown in FIG. 5. The raw audio FIG. 6ais portrayed in a representative spectral plot of a representativetelevision audio segment and the post equalized audio is portrayed inFIG. 6b where the audio is enhanced per the filter parameters of FIG. 5.The pre-emphasis process of 403 enhances the dynamic range of certaincoefficients and thus improves the quantization process 408 of thecoefficients. Data is then divided into frames of 20 ms and overlappedwith 50% of the previous frame as depicted in FIG. 7. The frame audio isthen shaped with a triangular window function 805 as depicted in FIG. 8with a resulting spectral distribution of 806. The next step in theprocess is autocorrelation of the framed audio 405 then the LPC process406 is applied whose coefficients are further transformed by the ISFfunction of 407 which are then normalized by a weighting functionsimilar to 1001 of FIG. 10 in step 408 which also includes the step ofquantization. Data is then framed into cue sets 409 and sent to theaudio matching system 410 for either enrollment of the reference audioinformation or for identification process of unknown media segments.

In the preferred embodiment of the invention, Linear Predictive Coding(LPC) is utilized for the primary step of coefficient generation butalternate embodiments include: Mel-Frequency Cepstral Coefficients(MFCC), Modified Discreet Cosine Transforms (MDCT), and/or Waveletsamong others. FIG. 20 represents a block diagram matrix of variousalternatives available to the invention for transforming audio intocoefficients useable by the invention. Said matrix maps four families2002, 2003, 2004, 2005 of possible algorithm combinations suitable foraudio transformation into coefficient frame output for usefulexploitation by the invention. Processes chain 2002 includes fourvariations from a common base of Autocorrelation 2002 a applied to theaudio signal 2001. Autocorrelation can directly provide one of the fouroutputs of coefficients 2017. The second process of the 2002 familyapplies Linear Predictive Coding (LPC) 2006 to the output of 2002 a tooutput LPC coefficients at 2009. Alternatively, said LPC 2006 values canbe further transformed by means of either LSP 2007 or ISF 2008 tofurther transform the coefficients. In all four cases coefficientoutputs are further processed by means of one of two possiblequantizations step of 2014 or 2015. The second family of processing isthe Mel Frequency Cepstral (MFC) Coefficient process begins with thetaking of Log value 2003 of the audio then further processing by meansof the MFC process 2010 prior to the final quantization step of either2014 or 2015. The Wavelet 2004 transform can be used with a suitablecoefficient generation step 2011 and finally the Modified DiscreetCosine Transform 2005 process can produce candidate cue sets(coefficient frames) by means of direct Coefficient Generation 2012 orby means of Bit Derivation (2013) producing a Hash String output. In allbut output 2013, the coefficients are quantized by either a linearprocess 2014 or by means of vector quantization 2015 then output fromthe system at 2016. In all of these processes, audio is transformed intohigh-entropy coefficient sets representing frames of audio with nearstationary power spectrum for the duration of the audio frame hencegenerating coefficient that can be appropriately hash indexed andapplied to a search and scoring means of Path Pursuit providing thepotential for accurate and continuous identification of audio segments.

FIG. 13 is a graph of the coefficients of the LPC process as poles of aZ-plane process represented by X's 1302. The transformation of LPCcoefficients to ISF coefficients results in zeros about the unit circle1301. FIG. 14 is a graph of the ISF coefficients over time illustratingtheir high entropy and hence suitability for a path pursuit-likematching process. It should be noted that in another embodiment of theinvention, the audio conversion process of the invention can functionutilizing only LPC output coefficients and not employing the step ofconversion to LSP or equivalent ISF coefficients as this LSP/ISF stepwas developed in prior art primarily for improving audio quality invocoder applications. It has been found that certain improvements inaudio quality may not measurably improve the accuracy of an audiomatching system.

FIG. 15 shows the formation of an audio cue data set from thecoefficient data 1507 with the addition of program time code 1508 andcertain program identification information also known as metadata 1509.Once formed, in FIG. 16 the audio cue 1601 is supplied to the mediasearch database where it is processed by an Audio Hash Function 1602creating hash key 1603 for storage in a search database 1604 where thehash key causes similar audio data cues to be grouped nearby to minimizesearch distance and hence improve overall system efficiency.

The client side of the invention is shown in FIG. 17 where a processsimilar to the enrollment function is generated in the client device1701. Audio from said client device is process into audio cues 1705 withthe addition of the local time 1707, also known as “wall time”, added tothe cues to provide relative time differences between cues. FIG. 18shows the unknown data cue addressing the reference media database bymeans of the same hash function as used to address said database duringthe enrollment process of the reference media. One or more candidates1802 are recovered from the database to be supplied to the matchingprocess 1803 as described above. Candidates are evaluated using linearalgebraic functions for selecting candidate data by means of evaluatingEuclidian distance in high-dimension space such as by means of ProbablePoint Location in Equal Balls (PPLEB), a process also known as suspectselection. A further step in the process of likely candidate (suspect)selection is carried out by Time Discount Binning (TDB) for a knownperiod. FIG. 19 shows the candidate (suspects) where each is representedby a bucket 1902 allocated after the process of harvesting of saidsuspect. Said buckets are leaky meaning the tokens have a preset timevalue and timeout which is the equivalent of a leaky bucket drainingover time. As unknown data cues arrive and more suspects are harvestedfrom the reference database, the number of tokens in the bucket thatidentifies the unknown cues will rise above a threshold value 1904 aftera period of three to ten seconds and thus identify the unknown data.This entire process can be understood by reference to the appendix ofinvention U.S. Pat. No. 8,595,781. An alternative means to scorecandidate matches can be achieved by the application of correlation ofsaid unknown cue 1801 to one or more candidate cues 1802. Said means ofcorrelation, not to be confused with auto-correlation as used herein, iswell known to the skilled person for finding the closest match of areference data item to one data item of a set of test data items. Hence,said scoring means by the process of mathematical correlation produces abest match by the identification system in place of time discountbinning. The process is further illustrated in FIG. 22a where each stepfrom Start 2202 a through 2206 a Within Range is similar to theprocesses leading to the above Time Discount Binning of FIG. 22. At step2207 a, the Correlation process is applied in placed of creating tokenbins. Step 2209 a selects the closest fit from the Correlation process2207 a. The winning values is further evaluated by 2211 a and, ifpositive, the candidate token identification is output as the result2212 a.

The process described above is one of many embodiments of the invention.The following description is the means of the invention by whichcoefficients are generated from the audio signal and common to mostembodiments.

The invention reveals that Linear Predictive Coded (LPC) coefficientsand their variants can be used in place of feature vectors orfingerprints for reliable detection of audio segments typically within afew seconds of analyzing unknown audio signals. The theory underlyingLPCs is well understood and practiced in signal communications systemsas a fundamental process of transcoding audio signals for packet-baseddigital communications systems. A subset of the common processes is usedfor the invention. The rationale behind the processes selected isprovided along with a detailed description of the many steps to producecoefficients beneficial for automated content recognition (ACR).

Referring again to FIG. 4, which illustrates a simplified block diagramof the process of processing audio from a TV Audio 401 source; is shouldbe understood that the audio signal processing steps 402 to 409 throughto the application of processed audio to an Audio Matching System 410 isthe same for the enrollment process of adding known audio segment cuesto a reference database 307 of FIG. 3 as it is for processing audiofrom, for example, a client smart TV and submitting said audio segmentcues via a network, such as the Internet, to said Audio Matching System410 for determining the identity of said unknown segments of cue values.

In more detail of the many steps to applying said audio representationsto an Audio Matching System 410, certain necessary Pre-Processing 402steps are applied to the audio which may include stereo to monauralconversion, down or up-sampling of the audio followed by Pre-emphasis(whitening) 403 then Framing, Shaping and Overlapping 404 where theaudio is segmented into frames of 20 to 100 milliseconds then a trianglewindow function 805 of FIG. 8 is applied to the signal of each frame,such as 701 of FIG. 7, to mitigate the abrupt start and stop of thesignal within the frame boundary. The final step of 404 is theoverlapping of frames by, in this embodiment, 50%. The overlapping istypically achieved by, in the current example of 50%, as seen by 701 to704 of FIG. 7, by starting a next audio frame at the half-way point ofthe audio of the previous frame such that the first one-half of the nextframe is the same audio as the last one-half of the previous frame, andso on. This process accommodates alignment differences between thereference database of known audio segments and the unknown audiosegments as received by the matching system server means 306 of FIG. 3.The pre-processed digital audio is the passed through an Autocorrelationprocess 405 in preparation for the conversion to the Linear PredictiveCoding (LPC) process 406. As audio passes through the block 406, it isevaluated by the Z-plane transform 1/A(z). The key to usefulness of thisprocess in matching unknown audio segments to a reference audio segmentdatabase, lies in the fact that the LPC transforms the time-domain audiointo a power spectral representation in the frequency domain, much likea Fourier Transform but in a Laplacian mode. Hence, the resultingtransformed audio information is quasi-stationary with respect to itspower spectral density, remaining relatively unchanged for at least tensof milliseconds. The transfer function 1/A(z) is an all-polerepresentation of the full-bandwidth audio transfer function. A(z) is aset of coefficients for a polynomial in the z domain where z representsê(−iwt). In a preferred embodiment, for wideband audio encoding, a 16thorder LPC (LPC 16) polynomial is employed. Higher-order polynomials canbe employed up to at least LPC 48. Higher-order polynomials are furtheradvantages by the application of the audio-band re-Emphasis 403 whenapplied to the audio prior to the LPC process. Yet a further improvementof relative high entropy distribution among the coefficients is theapplication of an LP Weighting function such as 1001 as applied to arepresentative LP Spectrum 1002 of FIG. 10. In one embodiment of theencoder, segments of audio of 20 milliseconds in duration are analyzedand converted into a set of 16 coefficients representing the channelinformation of, for example, an audio signal with a bandwidth of 8 kHz.In another embodiment of the encoder, segments of audio of 100milliseconds and a frequency bandwidth of 16 kHz are converted into setsof 48 coefficients. FIG. 5 shows an exemplary pre-emphasis filterproviding pre-emphasis to the audio prior to processing by the LPCtransform. FIG. 6 (A) shows the spectral characteristics of the audioprior to the pre-emphasis and (B) shows the audio spectrum post thepre-emphasis step. The particular filter of FIG. 5 affords a +15 dBboost of the frequencies from 1 kHz to the top of the audio band whichis 16 kHz in this embodiment.

The continuous frames of coefficients generated by the LPC process ofthe invention can serve in place of fingerprints, as used in the priorart, for an audio matching means where the processes of Path Pursuitprovides the matching mechanism. When the LPC process is used in audiovocoders, such as for audio communications, said LPC's excitationencoding sub-process provides two values per each 20 millisecond framewhich are a codebook representation of the waveform and an amplitude ofthe signal. An iterative algorithm is used to convert said excitationinto the codebook and is computationally large (expensive).Comparatively small changes in codebook values result in largeimprovements in perceived speech quality and, hence, the process is ofvalue to audio communications systems. However, for audio matchingsystems, small differences in codebook values do not result in the largeEuclidian distances between coefficients desired for audio matchingapplications. Due to the large processing demands and sub-optimaldistance characteristics of the codebooks, said excitation parameters donot benefit the invention and are, hence, not used.

In one embodiment, the LPC coefficients are not used directly from theoutput of the 1/A(z) model. Audio codecs for typical audiocommunications have led to computationally efficient processing means.In a widely used embodiment, the LPC coefficients are calculated usingan iterative algorithm using forward and backward prediction calledLevinson-Durbin. An appealing attribute of this method is thatreflection coefficients are easily derived as a byproduct. Thesecoefficients are used to produce lattice filters for the synthesis andprediction filters. This filter topology also offers robust performancewith low sensitivity to coefficient precision which is an usefulattribute also for audio matching systems.

Hence, the invention does not require all of the steps used for voicecommunications applications of LPC and thus useful coefficients can beproduced by means of a subset of said steps. In one embodiment, anexample of the reduced steps follows:

-   -   Capture 320 audio samples, 20 milliseconds at 16 kHz sample rate        (SR)    -   Or, capture 320 audio samples, 10 milliseconds at 32 kHz SR    -   Or, capture 2400 audio samples, 50 milliseconds at 48 kHz SR    -   There is no need for high-pass filter, typically set at 50 Hz,        as this process is already done on TV audio before transmission    -   Perform pre-emphasis of 4 kHz HPF resulting in a boost of +25 dB        at 16 kHz    -   Perform a 50% overlapping of audio frames    -   Auto-correlation on the audio outputs 16, 32 or 48 coefficients    -   Levenson-Durbin calculates 16 or 32 or 48 LPC coefficients

The audio inputs from a typical source as found in, for example, a smartTV, are stereo and are transmitted at a sample rate of 48 kHz. Forprocessing sample rates less than the received rate of 48 kHz, audiodown-conversion is performed by low-pass filtering to eliminatefrequency components above the Nyquist frequency which is two times thefrequency of interest, followed by a decimation process to convert saidaudio down to the desired sample rate. For example, to convert from 48kHz to 16 kHz requires a low-pass filter to eliminate frequencycomponents above 8 kHz. The filter output is then decimated by a factorof three to convert to the lower sample rate of 16 kHz. It is alsoobvious that, for automated content recognition, stereo inputs are notnecessary for good audio detection. The stereo input is thereforeconverted to monaural by combining the left and right channels, oralternatively, either the left or right channel can be used as a solerepresentative monaural channel.

To improve the distribution of the power spectrum, a whitening filter isthen added to the data path of the invention. Said filter boostsfrequencies above 4 kHz by up to 20 dB at the top frequency. Each 20milliseconds of audio (320 samples at 16 kHz) is packaged as a frame.

A simple triangle windowing function is applied to each audio frame toprepare the audio frames for LPC processing. Frame shaping is needed toreduce spurious signal generation at the edges due to the abrupt startand stop of the signal in each frame. Typically, a Hamming-like functionis employed to maximize audio fidelity. However, as fidelity of encodingis not important to the process of media identification, a simpletriangle function is all that is needed for the invention.

Levenson-Durbin calculates the LPC coefficients using theautocorrelation of the audio samples for input to the LPC function.Levenson-Durbin is used to calculate 16 coefficients in addition to aleading “1’ for a total of 17 values per frame from 17 autocorrelationlags, 0 −16. The details of said coding are well known to the skilledperson. Because a DC component is not present in the audio, as discussedabove, the autocorrelation function is equivalent to the covariance ofthe signal. Inversion of the covariance matrix results in an all-polerepresentation of the signal channel. Any method of matrix inversionsuch as Gauss elimination or Cholesky decomposition can be used. Thematrix is by definition real-valued and symmetric about the diagonal,also known as a Toeplitz matrix. Levenson-Durbin uses iterativeforward/backward estimation recursively to calculate the roots. Thisapproach is used almost universally in LPC analysis. Not only is saidapproach numerically stable and computationally efficient, but it alsoprovides the reflection coefficients as a by-product with little extracomputation. A lattice filter representation of the channel using thereflection coefficients is especially well suited to fixed-pointimplementation and is used throughout the general purpose vocoder andcan be beneficially employed by the invention. Shown in FIG. 9 in oneembodiment of the invention are the autocorrelation coefficients takenfrom a 20 millisecond audio segment. The FIG. 11 shows the LPCcoefficients calculated from the autocorrelation values.

In another embodiment, it may be found to be beneficial to follow theLPC process with further processing in the form of the conversion ofsaid LPC coefficients to either Line Spectral Pairs (LSP) or theequivalent Immittance Spectral Frequencies (ISF) as shown in FIG. 12.The IFS are derived from the LPC coefficients by first creatingsymmetric and anti-symmetric functions f_(1′) and f_(2′) of the sameorder as the LPC filter from the LPC coefficients:

f _(1′)(z)=A(z)+z ⁻¹⁶ A(z ⁻¹) and f _(2′)(z)=A(z)−z ⁻¹⁶ A(z ⁻¹)

The roots of these two equations lie on the unit circle and are theISFs. Like the LPC coefficients, the roots of f1 and f2 are conjugatesymmetric and only those on the upper half of the unit circle need to beevaluated. Exploiting this symmetry, two new functions f1 and f2 arecreated. F1 simply consists of the first 8 coefficients of f1′. F2consists of the first 7 coefficients of f2′ filtered using a differenceequation to remove the roots at 1 and −1. The roots of f1(z)=0 andf2(z)=0 are the ISFs. The roots of these functions can be found usingclassical methods such as Newton-Raphson or LaGuerre polynomials.However, due to special characteristics of these polynomials, acomputationally efficient approach using Chebyshev polynomials may beused.

Using the approach above, f1 and f2 for the LPC coefficients for theexample are shown in FIG. 14. The zero crossings of f1 and f2 are theISFs. The x axis corresponds to theta, the angle on the unit circle with0=0 degrees and 100=180 degrees. F1 and f2 are evaluated using only thereal component. For example, at x=10, the angle is 18 degrees and theinput to f1 and f2 is cosine (18*100/(2*pi))=0.95106. The zero crossingsare the ISF locations, with the ISF=cosine (theta). The first and lastzero crossings are roots of f1 and the roots alternate between f1 andf2. An efficient zero crossing detection algorithm was written whichexploits these properties to minimize the processing required. FIG. 13shows the LPC coefficients generated by the Levinson-Durbin algorithm asX's and the resulting ISFs as O's.

A plot over time of said ISF coefficients is found in FIG. 14illustrating a desirable entropic nature of the coefficients which islargely independent of the underlying audio signal from which saidcoefficients were indirectly derived. Is should be understood that theLPC coefficients will appear in a plot with similar shape.

It is interesting to note that the reflection coefficients and the ISFsare derived from the autocorrelation coefficients by a series of lineartransformations. Although there are divisions in the Levinson-Durbinalgorithm and division is not a linear process, they are used only forscaling and, thus, can be construed as multiplicative which is linear.As proof, if omitted from a double precision floating pointimplementation, the result will be the same. The observation isimportant because it suggests that the statistical properties of theautocorrelation, LPC coefficients, reflection coefficients, and the ISFsshould be very similar. Hence, in yet another embodiment of theinvention, the system of the invention can perform automated contentrecognition of audio content creating coefficient from just theautocorrelation data and not the LPC and not the ISF processes yetfurther improving the efficiency of the overall ACR system.

It should be understood from the above detailed description that theinvention provides a means to convert audio information intosemi-stationary frames of audio coefficients useful for the enrollmentand identification data of an automated content recognition system. Saidprocess provides the ability to continuously match audio informationfrom a very large population of audio sources such as smart TVs. Withappropriate central server scaling, said population could include tensof millions of devices. In addition, said audio ACR system can beefficiently combined with a video matching system such as taught byNeumeier and Liberty in U.S. Pat. No. 8,595,781 where both audio andvideo matching processes can share a common central processingarchitecture such as the path pursuit means of Neumeier. The inventionis distinct from the prior art in not employing a fingerprinting meansfor identification of audio and is more accurate with few false positiveresults and at the same time much more scalable such that it can beutilized for continuous identification of media and at the same timerequire a minimum of processing overhead at each client device.

FIG. 23 illustrates a system and/or an operational flow 2300representing example operations related to continuous audio matching. InFIG. 23 and in following figures that include various examples ofoperational flows, discussion and explanation may be provided withrespect to the above-described examples of FIGS. 1 through 22, and/orwith respect to other examples and contexts. However, it should beunderstood that the circuitry, means and/or operational flows may beexecuted in a number of other environments and contexts, and/or inmodified versions of FIGS. 1 through 22. Also, although the variousoperational flows are presented in the sequence(s) illustrated, itshould be understood that the various procedures carried out by theoperational flows may be performed in other orders than those which areillustrated, or may be performed concurrently. “Operational flow” asused herein may include circuitry for carrying out the flow. Aprocessing device, such as a microprocessor, may, via execution of oneor more instructions or other code-like appurtenances, become “circuitryconfigured for” a particular operation. An operational flow as carriedout by a processing device would render the processing device “circuitryconfigured for” carrying out each operation via execution of the one ormore instructions or other appurtenances.

After a start operation, the operational flow 2300 moves to operation2310. Operation 2310 depicts maintaining a reference match databaseincluding at least one coefficient corresponding to at least one audioframe of at least one ingested content and at least one contentidentification corresponding to the at least one ingested content. Forexample, as shown in and/or described with respect to FIGS. 1 through22, content is supplied to a media ingest operation which produces audioand/or video cue data and provides associated metadata (for example,identification of the received content such as a title, episode, orother identifier). The audio and/or video cue data is stored in adatabase along with the corresponding identification in real-time (i.e.as the content is received). The audio and/or video data is transformedinto values using a particular algorithm, function, and/or set offunctions. That particular algorithm, function, and/or set of functionsis also used by the client device as it processes audio and/or videodata. As the same point in the program content is processed at theingest operation and at the client device, the resulting audio and/orcoefficients will be the same or nearly the same due to the use of thesame algorithm, function, and/or set of functions by both the ingestoperation and the client device. Rather than storing the entirety of theprogram content, or just the audio portion of the program content, aframe of audio content is transformed into the much smaller coefficientand stored in conjunction with the identifier. The coefficient would notbe able to produce the audio, but would contain sufficient data to bematched with a corresponding coefficient sent by a client device inorder to retrieve the associated content identification from thereference match database.

Then, operation 2320 depicts receiving at least one transmission from atleast one client device including at least one client coefficientcorresponding to at least one audio frame renderable by the at least oneclient device. For example, as shown in and/or described with respect toFIGS. 1 through 22, as audio and/or video is able to be rendered by theclient device (i.e. played over the speaker or other audio output of theclient device), the audio and/or video data is transformed at the clientdevice into a coefficient using the same algorithm, function, and/or setof functions used by the ingest operation (not necessarily at the samerate as described elsewhere herein). The resulting coefficient istransmitted, usually via the Internet, to a matching server system whichcan access the reference match database.

Then, operation 2330 depicts identifying at least one content associatedwith the at least one client device at least partially based onsearching the reference match database using the at least one clientcoefficient as a search term. For example, as shown in and/or describedwith respect to FIGS. 1 through 22, the matching server system may use areceived coefficient from a client system to retrieve a suspect from thereference match database. A plurality of successive receivedcoefficients is used to retrieve multiple suspects, which are placed inbins correlating to possible program matches. Time discount binning isused through successive database retrievals to determine and/or identifythe most likely program being rendered by the client device. Theoperational flow then proceeds to an end operation.

FIG. 24 illustrates alternative embodiments of the example operationalflow 2300 of FIG. 23. FIG. 24 illustrates an example embodiment whereoperational flow 2310 may include at least one additional operation.Additional operations may include operation 2410, 2420, 2430, 2440,2450, and/or 2460.

Operation 2410 illustrates obtaining at least one real-time feed of atleast one broadcast of at least one content. For example, as shown inand/or described with respect to FIGS. 1 through 22, the matching serversystem may retrieve, via a satellite downlink of a network's nationwidebroadcast facility, a program. The matching server system may bereceiving the contents of multiple channels at once. By downlinkingdirectly from the network's nationwide broadcast facility, the matchingserver system receives the content in advance of the client devices, dueto client latency introduced by additional downlink and retransmissionoperations by local affiliates, cable operators, network head-ends, etc.

Then, operation 2420 illustrates encoding at least one audio sample ofthe at least one real-time feed. For example, as shown in and/ordescribed with respect to FIGS. 1 through 22, audio data for one or manychannels is converted to a stream of coefficients for storage in thereference media database. A continuous audio waveform is sampled into aplurality of frames which may occur at, for example, 50 times a secondor 20 ms frames. The sample rate is selected to maintain an effectivelystationary power spectral density of the audio information within thesample. In some embodiments, overlapping of adjacent audio frames isperformed to make up for any mismatch between start times of audiomatching by the matching server system and client device. The frame datais then transformed using functions which repeatably result in the samecoefficient value as would occur if the audio data were transformed atthe client device.

Then, operation 2430 illustrates storing the encoded at least one audiosample in association with the at least one content identification. Forexample, as shown in and/or described with respect to FIGS. 1 through22, the coefficient may be stored along with an indication of the nameof a program obtained via the ingest arrangement (e.g. satellite feed).The data is stored in a manner to facilitate retrieval of the data by apath pursuit means incorporating leaky buckets and time discount binningof results of successive data retrieval operations.

Operation 2420 may include at least one additional operation. Additionaloperations may include operation 2440. Operation 2440 illustratestransforming the at least one audio sample to the at least onecoefficient, the transforming at least partially based on at least onenormalization capable of repeatably providing coefficients associatedwith ingested audio content uncorrelated with specific frequencies. Forexample, as shown in and/or described with respect to FIGS. 1 through22, the transform process may include algorithms and/or functionsdesigned to “spread out” the coefficient values along a range of valuesin order to maximize the use of the entire range, to make the dataappear highly entropic. Without this spreading, coefficients would tendto congregate near a single point along the range of possible values forthe coefficients. For example, consider dialogue including a speakerwhose voice characteristics include a tone corresponding to a particularfrequency. Without the foregoing transformations designed to make thedata appear highly entropic, coefficients corresponding to the speakerwould tend to gather around one value corresponding to that frequency.Through application of functions disclosed herein, the coefficientsinstead are spread around their range of possible values, making themappear highly entropic and eliminating any relation of the resultingcoefficient to a particular audio frequency. Yet the functions arerepeatable in that two different systems (e.g. the matching serversystem and a client device) operating on the same audio content willoutput the same or nearly the same coefficient values (note that they donot need to be exactly the same because the subsequent time-discountbinning which establishes a likelihood of a match among multiplesuspects allows for slight variation in the coefficients correspondingto the same portion of the content).

Operation 2450 illustrates maintaining a reference match databaseincluding at least storing the at least one coefficient corresponding toat least one audio frame using locality sensitive hash indexing. In someembodiments, as shown in and/or described with respect to FIGS. 1through 22, for speedy retrieval of the data a number of mostsignificant bits may indicate a particular database server on which thecoefficient and program identification should be stored.

Operation 2460 illustrates maintaining at least two reference matchdatabases, including at least one audio reference match database and atleast one video reference match database, the system capable ofutilizing either the at least one audio reference match database or theat least one video reference match database to independently identifythe at least one content associated with the at least one client devicein response to receiving either at least one client coefficientcorresponding to at least one audio frame renderable by the at least oneclient device or at least one client coefficient corresponding to atleast one video sample renderable by the at least one client device. Insome embodiments, as shown in and/or described with respect to FIGS. 1through 22, a system may receive video ingest in addition to audioingest, facilitating identification of a program using either or both ofa stream of audio coefficients and/or a stream of video coefficients,which may serve to provide more robust matching by confirming anidentification made using audio coefficients using the videocoefficients, or providing an ability to switch between audio and videomatching as needed if the signals are interrupted.

FIG. 25 illustrates alternative embodiments of the example operationalflow 2300 of FIG. 23. FIG. 25 illustrates an example embodiment whereoperational flow 2320 may include at least one additional operation.Additional operations may include operation 2510, 2520, 2530, and/or2540.

Operation 2510 illustrates receiving at least one transmission from atleast one client device, the at least one client device including one ormore of at least one television, at least one smart television, at leastone media player, at least one set-top box, at least one game console,at least one NV receiver, at least one Internet-connected device, atleast one computing device, or at least one streaming media device. Forexample, as shown in and/or described with respect to FIGS. 1 through22, a widget may operate on the client device to transform an audiostream renderable on the client device into a stream of coefficients forsending to a matching server system. Many client devices render contentand have the ability to perform data processing tasks simultaneously. Insome instances the client action can occur on a smart television; indifferent embodiments the client action occurs on a set-top box (a cableor satellite receiver, e.g.) which receives the content and provides itto a television for playback.

Operation 2520 illustrates receiving at least one transmission streamfrom at least one client device, the at least one transmission streamincluding at least one sequence of client coefficients associated withone or more of at least one audio frame or at least one video framerenderable by the at least one client device to identify at least onecontent renderable by the at least one client device, the at least onesequence including at least some audio client coefficients. For example,as shown in and/or described with respect to FIGS. 1 through 22, theclient device of the invention sends coefficients corresponding tosamples of the audio content to the matching server system, thegeneration of coefficients and sending occurring at a particularinterval (which may be periodic or aperiodic and can be alteredmid-stream). The client device may additionally send coefficientsproduced using pixel data from the content received by the clientdevice, but the invention disclosed herein at least sometimes sendsaudio coefficients irrespective of whether video coefficients are sent.

Operation 2530 illustrates receiving at least one transmission from atleast one client device including at least one client coefficientcorresponding to at least one audio frame renderable by the at least oneclient device, the at least one client coefficient corresponding to atleast one audio frame renderable by the at least one client devicedetermined at least partially via at least one transform identical to atleast one transform utilized in maintaining the reference matchdatabase. For example, as shown in and/or described with respect toFIGS. 1 through 22, the client device uses the same transform functionas is utilized by the matching server system (although not necessarilyas the same rate as disclosed elsewhere herein) to obtain coefficientscorresponding to audio content about to be played over the speaker oraudio out of the client device. The two systems using the same transformmean that at the same point in the program content, the resultingcoefficient values produced by the client device and the matching serversystem will be substantially the same (subject to the overlap functionwhich aligns audio frames in instances where the framing begins at adifferent time offset on the two systems).

Operation 2540 illustrates receiving at least one transmission from atleast one client device including at least one client coefficientcorresponding to at least one audio frame renderable by the at least oneclient device, the at least one client coefficient corresponding to atleast one audio frame renderable by the at least one client devicedetermined at least partially via sampling at least one audio streaminto one or more frames and overlapping the one or more frames previousto normalization of the overlapping one or more frames. For example, asshown in and/or described with respect to FIGS. 1 through 22,overlapping of the frames aligns the audio frames in instances where theframing begins at a different time offset on the client device than itdid on the matching server system which could occur when, for example,the client device is tuned to a new channel in the middle of a programbeing broadcast.

FIG. 26 illustrates alternative embodiments of the example operationalflow 2300 of FIG. 23. FIG. 26 illustrates an example embodiment whereoperational flow 2330 may include at least one additional operation.Additional operations may include operation 2610, 2620, 2630, 2640,2650, and/or 2660.

Operation 2610 illustrates utilizing one or more video coefficientsreceived from the at least one client device for obtaining one or moresuspects from a reference match database associated with videocoefficients. For example, as shown in and/or described with respect toFIGS. 1 through 22, a path pursuit algorithm obtains a plurality ofsuspects corresponding to successive video coefficients received by thematching server system. Video matching may function provided that theclient device is producing an unaltered display of the content;activation of an on-screen menu or television zoom mode, or an on-screengraphic such as a watermark added by a local broadcaster, may cause thevideo matching to fail.

Then, operation 2620 illustrates detecting one or more media contentalterations from the at least one client device. For example, as shownin and/or described with respect to FIGS. 1 through 22, the matchingserver system may detect that a probability of a particular binidentifying a correct program is below a particular threshold to declarea particular bin the likely content-identifying bin. This could occurwhen received video coefficients sent while an on-screen channel guideis active insufficiently match coefficients in the database.Alternatively, the widget of the client device could detect theactivation of the on-screen channel guide and initiate transmission ofthe audio coefficients or signal the matching server system of theactivation.

Then, operation 2630 illustrates switching content identification toutilizing one or more audio coefficients received from the at least oneclient device for obtaining further suspects from a reference matchdatabase associated with audio coefficients. For example, as shown inand/or described with respect to FIGS. 1 through 22, the matching serversystem upon interference with video matching occurring (detection and orsignaling relating to the on-screen channel guide, for example) canswitch to using matching with the audio coefficients, because the audiosignal is typically not interrupted by an on-screen channel guide, oradded watermark, or other interference with on-screen video (i.e. mediacontent alteration).

Operation 2620 may include at least one additional operation. Additionaloperations may include operation 2640 and/or operation 2650.

Operation 2640 illustrates receiving at least one indication of at leastone of an on-screen graphic, a fade to black, or a video zoom modeassociated with the at least one client device. For example, as shown inand/or described with respect to FIGS. 1 through 22, as discussed above,the matching server system may detect a particular media contentalteration such as an on-screen graphic, a fade to black, or a videozoom mode which would interfere with matching using video coefficients.Such detection may take place when the content matching is unable tomatch a program with sufficient certainty, likelihood, and/orprobability. Alternatively, a client device could signal the matchingserver system that a media content alteration such as a zoom mode isoccurring. Such a signal could cause the matching server system to beginusing the audio coefficients.

Then, operation 2650 illustrates signaling to switch to audio contentidentification at least partially based on the at least one indication.In some embodiments, as shown in and/or described with respect to FIGS.1 through 22, in instances where video matching is not working, thesystem may switch to identification using audio coefficients. In someinstances, leaky buckets created in association with video matching arere-created and time discount binning begins anew upon the switch toaudio matching. In other instances, the content matching operationleaves the suspects from the video matching in the existing bins andbegins adding suspects from the audio matching to the bins such that, inthe time intervals immediately following the switch to audio, a bin mayhave both video suspects and audio suspects, where the video suspectsmay leak from the buckets first but both video and audio suspects willbe used to declare an identification.

Operation 2660 illustrates determining at least one identification ofthe at least one content associated with the at least one client deviceat least partially based on time-discount binning one or more suspectsretrieved from the reference match database using the at least oneclient coefficient corresponding to at least one audio frame renderableby the at least one client device. In some embodiments, as shown inand/or described with respect to FIGS. 1 through 22, upon receipt of anaudio coefficient from a client device, it is used as a search query ofthe reference media database. One or more suspects corresponding to theaudio coefficient are retrieved, each of which linked to a particularprogram identifier. The suspects are placed in bins assigned toparticular programs. The process is repeated with each successivereceived audio coefficient and a bin receiving the most suspects mostlikely corresponds to the program being viewed. The oldest suspects areremoved over time (i.e. the “leaky buckets”) and when a channel ischanged on the client, suspects begin going in different bins responsiveto the different audio coefficients produced by the channel change.

FIG. 27 illustrates an alternative embodiment of the example operationalflow 2300 of FIG. 23. FIG. 26 illustrates an example embodiment whereoperational flow 2310 may include at least one additional operation 2710and where operational flow 2330 may include at least one additionaloperation 2720.

Operation 2710 illustrates storing one or more transformed powerspectral coefficients associated with at least one audio portion of theat least one ingested content in associated with the at least onecontent identification. For example, as shown in and/or described withrespect to FIGS. 1 through 22, the media ingest operation's audiocoefficients begin as frames of ingested audio content during sampleshaving a frame size small enough that the power spectral densitycorresponding to the ingested audio signal remains effectively constantthroughout the sample. The frame is transformed using operationsdisclosed herein to data subsequently stored in the reference mediadatabase and associated with an identification of a program beingingested.

Then, operation 2720 illustrates time-discount binning one or moresuspects obtained from the reference match database, the obtaining atleast partially based on one or more received transformed power spectralcoefficients associated with at least one audio content renderable bythe at least one client device. For example, as shown in and/ordescribed with respect to FIGS. 1 through 22, the client device sendingoperation's audio coefficients also begin as frames of audio content,these frames corresponding to an audio portion of a program being playedback on the client device, the frames obtained during samples having aframe size small enough that the power spectral density corresponding tothe audio signal of the program played back on the client device remainseffectively constant throughout the sample. Matching the coefficients ofthe known content being ingested to coefficients of the client deviceplaying back an unknown content will lead to identification of thecontent being played back by the client device.

FIG. 28 illustrates alternative embodiments of the example operationalflow 2300 of FIG. 23. FIG. 26 illustrates an example embodiment whereoperational flow 2300 may include at least one additional operation.Additional operations may include operation 2810, 2820, 2830, 2840,and/or 2850.

Operation 2810 illustrates continuously identifying the at least onecontent associated with the at least one client device at leastpartially based on continuously maintaining the reference matchdatabase, continuously receiving transmissions from the at least oneclient device, and continuously searching the reference match databaseusing client coefficients associated with subsequent transmissions assearch terms. For example, as shown in and/or described with respect toFIGS. 1 through 22, a received coefficient from a client device is usedas a search query for the reference media database, with the resultbeing used in a time discount binning operation. Subsequent coefficientsare received from the client device and used as subsequent databasesearches with the results being used in the time discount binningoperation. Given enough received audio coefficients from the clientdevice, a program identification is made. Should a channel be changed onthe client device, the stream of coefficients continues and a differentprogram identification may subsequently be made. Thus, the audiomatching is continuous audio matching, continuing even when a channel ischanged. The operational flow then proceeds to an end operation.

Operation 2820 illustrates maintaining a second reference match databaseincluding at least one coefficient corresponding to at least one videoframe of at least one ingested content and at least one contentidentification corresponding to the at least one ingested content. Forexample, as shown in and/or described with respect to FIGS. 1 through22, in addition to producing a stream of audio coefficients for storagein the reference match database during the ingest operation, a stream ofvideo coefficients may also be produced for storage in a reference matchdatabase corresponding to video. For optimal performance, the databasesmay be placed on different servers or server farms.

Then, operation 2830 illustrates altering a content identificationmethod related to the at least one client device, the altering a contentidentification method including at least one of switching from contentidentification based on video coefficients to content identificationbased on audio coefficients or switching from content identificationbased on audio coefficients to content identification based on videocoefficients. For example, as shown in and/or described with respect toFIGS. 1 through 22, the content identification operation may switchbetween matching using the audio coefficients and matching using thevideo coefficients as needed; for example, if an interruption in one ofthe audio or video occurs, the matching may switch to the other method.The operational flow then proceeds to an end operation.

Operation 2840 illustrates controlling the at least one client device,including at least signaling the at least one client device to switchfrom transmission of client coefficients corresponding to video framesto transmission of client coefficients corresponding to audio frames.For example, as shown in and/or described with respect to FIGS. 1through 22, if the content identification operation is unable toreliably choose an identification of a program based on a stream ofvideo coefficients from the client device, the matching server systemmay send a command over the Internet to the client device to beginsending audio coefficients instead of or in addition to the videocoefficients so that content identification may be attempted using theaudio coefficients. The converse is also possible (i.e. the matchingserver system may instruct the client to begin sending videocoefficients instead of or in addition to the audio coefficients). Theoperational flow then proceeds to an end operation.

Operation 2850 illustrates controlling the at least one client device,including at least signaling the at least one client device to transmitclient coefficients corresponding to audio frames at a particular rate.For example, as shown in and/or described with respect to FIGS. 1through 22, it is not necessary that the audio coefficients be sent bythe client device at the same rate as the rate at which they areproduced during ingest. The matching server system may instruct theclient device to send coefficients less frequently once an initialidentification is made. Alternatively, the matching server system mayinstruct the client device to send coefficients more frequently wherethe importance of an accurate and/or faster identification is greater.The operational flow then proceeds to an end operation.

Certain aspects of the present invention include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present inventioncould be embodied in software, firmware or hardware, and when embodiedin software, could be downloaded to reside on and be operated fromdifferent platforms used by real-time network operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus.

Furthermore, computers or computing means referred to in thespecification may include a single processor or may employmultiple-processor designs for increased computing capability.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may also be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description above.In addition, the present invention is not described with reference toany particular programming language or operating systems. It isappreciated that a variety of programing languages and operating systemsmay be used to implement the teachings of the present invention asdescribed herein.

The system and methods, flow diagrams, and structure block diagramsdescribed in this specification may be implemented in computerprocessing systems including program code comprising programinstructions that are executable by a computer processing system. Otherimplementations may also be used. Additionally, the flow diagrams andstructure block diagrams herein described describe particular methodsand/or corresponding acts in support of steps and correspondingfunctions in support of disclosed structural means, may also be utilizedto implement corresponding software structures and algorithms, andequivalents thereof.

Embodiments of the subject matter described in this specification can beimplemented as one or more computer program products, i.e., one or moremodules of computer program instructions encoded on a tangible programcarrier for execution by, or to control the operation of, dataprocessing apparatus. The computer readable medium can be a machinereadable storage device, a machine readable storage substrate, a memorydevice, or a combination of one or more of them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program can bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program can be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a suitablecommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

The essential elements of a computer are a processor for performinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto optical disks,or optical disks. However, a computer need not have such devices.Processors suitable for the execution of a computer program include, byway of example only and without limitation, both general and specialpurpose microprocessors, and any one or more processors of any kind ofdigital computer. Generally, a processor will receive instructions anddata from a read only memory or a random access memory or both.

To provide for interaction with a user or manager of the systemdescribed herein, embodiments of the subject matter described in thisspecification can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor,for displaying information to the user and a keyboard and a pointingdevice, e.g., a mouse or a trackball, by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well. For example, feedback provided to theuser can be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes back end component(s)including one or more data servers, or that includes one or moremiddleware components such as application servers, or that includes afront end component such as a client computer having a graphical userinterface or a Web browser through which a user or administrator caninteract with some implementations of the subject matter described isthis specification, or any combination of one or more such back end,middleware, or front end components. The components of the system can beinterconnected by any form or medium of digital data communication, suchas a communication network. The computing system can include clients andservers. A client and server are generally remote from each other andtypically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on therespective computers and having a client server relationship to eachother.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in standard integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies equally regardless of the particular type of signal bearingmedia used to actually carry out the distribution. Examples of a signalbearing media include, but are not limited to, the following: recordabletype media such as floppy disks, hard disk drives, CD ROMs, digitaltape, and computer memory; and transmission type media such as digitaland analog communication links using TDM or IP based communication links(e.g., packet links).

Those having skill in the art will recognize that the state of the arthas progressed to the point where there is little distinction leftbetween hardware and software implementations of aspects of systems; theuse of hardware or software is generally (but not always, in that incertain contexts the choice between hardware and software can becomesignificant) a design choice representing cost vs. efficiency tradeoffs.Those having skill in the art will appreciate that there are variousvehicles by which processes and/or systems and/or other technologiesdescribed herein can be effected (e.g., hardware, software, and/orfirmware), and that the preferred vehicle will vary with the context inwhich the processes and/or systems and/or other technologies aredeployed. For example, if an implementer determines that speed andaccuracy are paramount, the implementer may opt for a mainly hardwareand/or firmware vehicle; alternatively, if flexibility is paramount, theimplementer may opt for a mainly software implementation; or, yet againalternatively, the implementer may opt for some combination of hardware,software, and/or firmware. Hence, there are several possible vehicles bywhich the processes and/or devices and/or other technologies describedherein may be effected, none of which is inherently superior to theother in that any vehicle to be utilized is a choice dependent upon thecontext in which the vehicle will be deployed and the specific concerns(e.g., speed, flexibility, or predictability) of the implementer, any ofwhich may vary. Those skilled in the art will recognize that opticalaspects of implementations will typically employ optically-orientedhardware, software, and or firmware.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in standard integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies equally regardless of the particular type of signal bearingmedia used to actually carry out the distribution. Examples of a signalbearing media include, but are not limited to, the following: recordabletype media such as floppy disks, hard disk drives, CD ROMs, digitaltape, and computer memory; and transmission type media such as digitaland analog communication links using TDM or IP based communication links(e.g., packet links).

The herein described aspects depict different components containedwithin, or connected with, different other components. It is to beunderstood that such depicted architectures are merely exemplary, andthat in fact many other architectures can be implemented which achievethe same functionality. In a conceptual sense, any arrangement ofcomponents to achieve the same functionality is effectively “associated”such that the desired functionality is achieved. Hence, any twocomponents herein combined to achieve a particular functionality can beseen as “associated with” each other such that the desired functionalityis achieved, irrespective of architectures or intermedial components.Likewise, any two components so associated can also be viewed as being“operably connected”, or “operably coupled”, to each other to achievethe desired functionality, and any two components capable of being soassociated can also be viewed as being “operably couplable”, to eachother to achieve the desired functionality. Specific examples ofoperably couplable include but are not limited to physically mateableand/or physically interacting components and/or wirelessly interactableand/or wirelessly interacting components and/or logically interactingand/or logically interactable components.

While particular aspects of the present subject matter described hereinhave been shown and described, it will be apparent to those skilled inthe art that, based upon the teachings herein, changes and modificationsmay be made without departing from the subject matter described hereinand its broader aspects and, therefore, the appended claims are toencompass within their scope all such changes and modifications as arewithin the true spirit and scope of this subject matter describedherein. Furthermore, it is to be understood that the invention isdefined by the appended claims. It will be understood by those withinthe art that, in general, terms used herein, and especially in theappended claims (e.g., bodies of the appended claims) are generallyintended as “open” terms (e.g., the term “including” should beinterpreted as “including but not limited to,” the term “having” shouldbe interpreted as “having at least,” the term “includes” should beinterpreted as “includes but is not limited to,” etc.). It will befurther understood by those within the art that if a specific number ofan introduced claim recitation is intended, such an intent will beexplicitly recited in the claim, and in the absence of such recitationno such intent is present. For example, as an aid to understanding, thefollowing appended claims may contain usage of the introductory phrases“at least one” and “one or more” to introduce claim recitations.However, the use of such phrases should not be construed to imply thatthe introduction of a claim recitation by the indefinite articles “a” or“an” limits any particular claim containing such introduced claimrecitation to inventions containing only one such recitation, even whenthe same claim includes the introductory phrases “one or more” or “atleast one” and indefinite articles such as “a” or “an” (e.g., “a” and/or“an” should typically be interpreted to mean “at least one” or “one ormore”); the same holds true for the use of definite articles used tointroduce claim recitations. In addition, even if a specific number ofan introduced claim recitation is explicitly recited, those skilled inthe art will recognize that such recitation should typically beinterpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, typicallymeans at least two recitations, or two or more recitations).Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (e.g., “a system having at least one of A, B, and C”would include but not be limited to systems that have A alone, B alone,C alone, A and B together, A and C together, B and C together, and/or A,B, and C together, etc.). In those instances where a conventionanalogous to “at least one of A, B, or C, etc.” is used, in general sucha construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, or C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.).

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment.

Conversely, various features that are described in the context of asingle embodiment can also be implemented in multiple embodimentsseparately or in any suitable subcombination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination may be directed to a subcombination or variation ofa subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

PERTINENT TECHNICAL MATERIALS

-   KABAL (P.), RAMACHANDRAN (R. P.): The computation of line spectral    frequencies using Chebyshev polynomials, IEEE Trans. on ASSP, Vol.    34, No. 6, pp. 1419-1426, 1986.    -   ITAKURA (F.): Line spectral representation of linear predictive        coefficients of speech signals, J. Acoust. Soc. Amer., Vol. 57,        Supplement No. 1, S35,1975.-   BISTRITZ (Y.), PELLERM (S.): Immittance Spectral Pairs (ISP) for    speech encoding, Proc. ICASSP'93, pp. II-9 to II-12.-   Neumeier U.S. Pat. No. 8,595,781, METHODS FOR IDENTIF YING VIDEO    SEGMENTS AND DISPLAYING CONTEXTU AL TARGETED CONTENT ON A CONNECTED    TELEVISION-   Neumeier U.S. Pat. No. 8,769,584 B2, METHODS FOR DISPLAYING    CONTEXTUALLY TARGETED CONTENT ON A CONNECTED TELEVISION-   Neumeier U.S. Pat. No. 9,055,335—SYSTEMS AND METHODS FOR ADDRESSING    A MEDIA DATABASE USING DISTANCE ASSOCIATIVE HASHING-   Audible Magic U.S. Pat. No. 5,918,223, METHOD AND ARTICLE OF    MANUFACTURE FOR CONTENT-BASED ANALYSIS, STORAGE, RETRIEVAL, AND    SEGMENTATION OF AUDIO INFORMATION-   Civolution U.S. Pat. No. 8,959,202 B2-   Shazam U.S. Pat. No. 6,990,453-   Zeitera Audio Matching—Ser. No. 14/589,366 Application

I claim:
 1. A system, comprising: circuitry configured for maintaining areference match database including at least one coefficientcorresponding to at least one audio frame of at least one ingestedcontent and at least one content identification corresponding to the atleast one ingested content; circuitry configured for receiving at leastone transmission from at least one client device including at least oneclient coefficient corresponding to at least one audio frame renderableby the at least one client device; and circuitry configured foridentifying at least one content associated with the at least one clientdevice at least partially based on searching the reference matchdatabase using the at least one client coefficient as a search term. 2.The system of claim 1, wherein circuitry configured for maintaining areference match database including at least one coefficientcorresponding to at least one audio frame of at least one ingestedcontent and at least one content identification corresponding to the atleast one ingested content comprises: circuitry configured for obtainingat least one real-time feed of at least one broadcast of at least onecontent; circuitry configured for encoding at least one audio sample ofthe at least one real-time feed; and circuitry configured for storingthe encoded at least one audio sample in association with the at leastone content identification.
 3. The system of claim 2, wherein circuitryconfigured for encoding at least one audio sample of the at least onereal-time feed comprises: circuitry configured for transforming the atleast one audio sample to the at least one coefficient, the transformingat least partially based on at least one normalization capable ofrepeatably providing coefficients associated with ingested audio contentuncorrelated with specific frequencies.
 4. The system of claim 1,wherein circuitry configured for maintaining a reference match databaseincluding at least one coefficient corresponding to at least one audioframe of at least one ingested content and at least one contentidentification corresponding to the at least one ingested contentcomprises: circuitry configured for maintaining a reference matchdatabase including at least storing the at least one coefficientcorresponding to at least one audio frame using locality sensitive hashindexing.
 5. The system of claim 1, wherein circuitry configured formaintaining a reference match database including at least onecoefficient corresponding to at least one audio frame of at least oneingested content and at least one content identification correspondingto the at least one ingested content comprises: circuitry configured formaintaining at least two reference match databases, including at leastone audio reference match database and at least one video referencematch database, the system capable of utilizing either the at least oneaudio reference match database or the at least one video reference matchdatabase to independently identify the at least one content associatedwith the at least one client device in response to receiving either atleast one client coefficient corresponding to at least one audio framerenderable by the at least one client device or at least one clientcoefficient corresponding to at least one video sample renderable by theat least one client device.
 6. The system of claim 1, wherein circuitryconfigured for receiving at least one transmission from at least oneclient device including at least one client coefficient corresponding toat least one audio frame renderable by the at least one client devicecomprises: circuitry configured for receiving at least one transmissionfrom at least one client device, the at least one client deviceincluding one or more of at least one television, at least one smarttelevision, at least one media player, at least one set-top box, atleast one game console, at least one NV receiver, at least oneInternet-connected device, at least one computing device, or at leastone streaming media device.
 7. The system of claim 1, wherein circuitryconfigured for receiving at least one transmission from at least oneclient device including at least one client coefficient corresponding toat least one audio frame renderable by the at least one client devicecomprises: circuitry configured for receiving at least one transmissionstream from at least one client device, the at least one transmissionstream including at least one sequence of client coefficients associatedwith one or more of at least one audio frame or at least one video framerenderable by the at least one client device to identify at least onecontent renderable by the at least one client device, the at least onesequence including at least some audio client coefficients.
 8. Thesystem of claim 1, wherein circuitry configured for receiving at leastone transmission from at least one client device including at least oneclient coefficient corresponding to at least one audio frame renderableby the at least one client device comprises: circuitry configured forreceiving at least one transmission from at least one client deviceincluding at least one client coefficient corresponding to at least oneaudio frame renderable by the at least one client device, the at leastone client coefficient corresponding to at least one audio framerenderable by the at least one client device determined at leastpartially via at least one transform identical to at least one transformutilized in maintaining the reference match database.
 9. The system ofclaim 1, wherein circuitry configured for receiving at least onetransmission from at least one client device including at least oneclient coefficient corresponding to at least one audio frame renderableby the at least one client device comprises: circuitry configured forreceiving at least one transmission from at least one client deviceincluding at least one client coefficient corresponding to at least oneaudio frame renderable by the at least one client device, the at leastone client coefficient corresponding to at least one audio framerenderable by the at least one client device determined at leastpartially via sampling at least one audio stream into one or more framesand overlapping the one or more frames previous to normalization of theoverlapping one or more frames.
 10. The system of claim 1, whereincircuitry configured for identifying at least one content associatedwith the at least one client device at least partially based onsearching the reference match database using the at least one clientcoefficient as a search term comprises: circuitry configured forutilizing one or more video coefficients received from the at least oneclient device for obtaining one or more suspects from a reference matchdatabase associated with video coefficients; circuitry configured fordetecting one or more media content alterations from the at least oneclient device; and circuitry configured for switching contentidentification to utilizing one or more audio coefficients received fromthe at least one client device for obtaining further suspects from areference match database associated with audio coefficients.
 11. Thesystem of claim 10, wherein circuitry configured for detecting one ormore media content alterations from the at least one client devicecomprises: circuitry configured for receiving at least one indication ofat least one of an on-screen graphic, a fade to black, or a video zoommode associated with the at least one client device; and circuitryconfigured for signaling to switch to audio content identification atleast partially based on the at least one indication.
 12. The system ofclaim 1, wherein circuitry configured for identifying at least onecontent associated with the at least one client device at leastpartially based on searching the reference match database using the atleast one client coefficient as a search term comprises: circuitryconfigured for determining at least one identification of the at leastone content associated with the at least one client device at leastpartially based on time-discount binning one or more suspects retrievedfrom the reference match database using the at least one clientcoefficient corresponding to at least one audio frame renderable by theat least one client device.
 13. The system of claim 1, wherein circuitryconfigured for identifying at least one content associated with the atleast one client device at least partially based on searching thereference match database using the at least one client coefficient as asearch term comprises: circuitry configured for utilizing one or morevideo coefficients received from the at least one client device forobtaining one or more suspects from a reference match databaseassociated with video coefficients; circuitry configured for associatingthe at least one client device with at least one content identificationat least partially based on the one or more suspects; and circuitryconfigured for confirming the association of the at least one clientdevice with at least one content identification at least partially basedon one or more audio coefficients received from the at least one clientdevice.
 14. The system of claim 1, wherein circuitry configured formaintaining a reference match database including at least onecoefficient corresponding to at least one audio frame of at least oneingested content and at least one content identification correspondingto the at least one ingested content comprises: circuitry configured forstoring one or more transformed power spectral coefficients associatedwith at least one audio portion of the at least one ingested content inassociated with the at least one content identification; and whereincircuitry configured for identifying at least one content associatedwith the at least one client device at least partially based onsearching the reference match database using the at least one clientcoefficient as a search term comprises: circuitry configured fortime-discount binning one or more suspects obtained from the referencematch database, the obtaining at least partially based on one or morereceived transformed power spectral coefficients associated with atleast one audio content renderable by the at least one client device.15. The system of claim 1, further comprising: circuitry configured forcontinuously identifying the at least one content associated with the atleast one client device at least partially based on continuouslymaintaining the reference match database, continuously receivingtransmissions from the at least one client device, and continuouslysearching the reference match database using client coefficientsassociated with subsequent transmissions as search terms.
 16. The systemof claim 1, further comprising: circuitry configured for maintaining asecond reference match database including at least one coefficientcorresponding to at least one video frame of at least one ingestedcontent and at least one content identification corresponding to the atleast one ingested content; and circuitry configured for altering acontent identification method related to the at least one client device,the altering a content identification method including at least one ofswitching from content identification based on video coefficients tocontent identification based on audio coefficients or switching fromcontent identification based on audio coefficients to contentidentification based on video coefficients.
 17. The system of claim 1,further comprising: circuitry configured for controlling the at leastone client device, including at least signaling the at least one clientdevice to switch from transmission of client coefficients correspondingto video frames to transmission of client coefficients corresponding toaudio frames.
 18. The system of claim 1, further comprising: circuitryconfigured for controlling the at least one client device, including atleast signaling the at least one client device to transmit clientcoefficients corresponding to audio frames at a particular rate.
 19. Asystem, comprising: means for maintaining a reference match databaseincluding at least one coefficient corresponding to at least one audioframe of at least one ingested content and at least one contentidentification corresponding to the at least one ingested content; meansfor receiving at least one transmission from at least one client deviceincluding at least one client coefficient corresponding to at least oneaudio frame renderable by the at least one client device; and means foridentifying at least one content associated with the at least one clientdevice at least partially based on searching the reference matchdatabase using the at least one client coefficient as a search term,wherein at least one of the means for maintaining, the means forreceiving, or the means for identifying is at least partiallyimplemented in hardware.
 20. A method, comprising: maintaining areference match database including at least one coefficientcorresponding to at least one audio frame of at least one ingestedcontent and at least one content identification corresponding to the atleast one ingested content; receiving at least one transmission from atleast one client device including at least one client coefficientcorresponding to at least one audio frame renderable by the at least oneclient device; and identifying at least one content associated with theat least one client device at least partially based on searching thereference match database using the at least one client coefficient as asearch term, wherein at least one of the maintaining, receiving, oridentifying is at least partially implemented in hardware.